Software testing, which verifies and documents that a piece of software is operating correctly, is an important component of software development, and is often the most costly part of the process. With software testing, a software developer or software tester, oftentimes a specialist member of the software development team, proves that the finished program or software product meets its design requirements, and will function satisfactorily when released for practical operation. At various stages during the coding of a software product or, more likely, a portion of a large software project (and especially when the coding is declared complete for the first time), a developer often stops to perform unit testing to check to make sure that the product is working as designed. Many of the same tests will be repeated at various stages of development and new tests will be added from time to time.
After the developer is satisfied with the finished software product, it is first turned over to feature testers who will run a set of tests often related only to the newly written parts of the software product, to confirm the developer's claim that the finished product is working exactly as stipulated by the requirements. Then, integration or system testers will run a much larger set of tests involving the entire software product to make sure that other pre-existing features of the software are still functioning properly as they did before, and that the old and new features work properly with each other. This larger set of tests is often compiled by adding the most important tests from feature testing to the previous set of system tests. This growing set of system tests is sometimes referred to as a collection of regression tests. As bugs are found, the developer will fix them to produce a new software version and the same set of regression tests will be repeated many times over.
Regression testing is a well-known verification technique that involves continuously re-testing the design throughout modification and implementation to ensure that program errors are corrected and that design modifications or features added to the program do not introduce new errors. In carrying out regression testing the same tests are repeated to ensure that the behavior of the program being tested remains unchanged. Typically, the results of the various regression tests simply indicate whether or not the tests pass or fail. At times, the program being tested is a smaller portion of the overall program project. Accordingly, the regression tests may have to include an initialization step to simulate the environment of the program under actual operating conditions. As time goes on, and the program being tested undergoes a number of modifications, initialization and updating of the regression tests can require substantial investment. Oftentimes, regression testing is one of the most expensive and critical aspects of software maintenance, and improvements in testing such as test automation are being sought to help control program development and maintenance costs.
One of the most effective automatic tools is an automatic regression test harness in which regression tests are executed in a defined sequence, generating supporting documentation that a software program meets its stated requirements. A test harness comprises a collection of tests designed to verify proper behavior of a program. A test establishes an environment for the program, invokes one or more portions of the program, and compares one or more results of the program to one or more expected results for the program. The one or more expected results may be hard-coded into the test, for automated execution at a later date. As one shortcoming, the expected results for the program must be known when designing the test. It is desirable for the test harness to be developed independently from the determination of the expected results. Improvements in the reduction of user development time are still being sought.
Automated test harnesses are typically written in one of the popular scripting languages, such as ksh, perl, or python. Lengthy sessions of complicated programming are typically required. Code has to be written to perform one or more of the following steps: emulating the test developer's input or other initialization procedures, recording or otherwise capturing the output resulting from execution of the test developer's commands, comparing the test run output to some objective or standardized answers, and tabulating the results of the tests and with an indication of whether or not the tests were successful. At times, it is desirable to add new tests to an existing test harness. As with other newly written code, the new tests need to be verified for accuracy and reliability. In several examples of commercial testing software, a high level model of the software is created. Essentially, this requires a time-consuming effort of skilled professionals and accordingly is obtained only at a substantial cost. Once the model is created and debugged, a template of the code for the software and a test harness can be generated. However, much of the software developed for commercial purposes does not lend itself to such an approach.
Even after software is developed, tested and approved, modifications such as adjustments to the program require corresponding modifications to the test harness. For example, a test designer spends time manually computing new expected results based on the adjustments to the program. The test designer must then manually update the expected values of the tests of the test harness with the new expected results. For example, the test designer must hard-code the new results into the test, which adds additional lines of code, increasing a margin of error for the test harness as well as increasing difficulty of debugging the test harness. Modifications to the software undergoing testing may also require substantial modifications to the tests of the test harness. Once new or modified test harnesses are developed, they must be debugged and approved after having proven that they can operate with a measure of confidence and reliability.
A need exists for providing a reduction in labor and other costly investments required to develop, operate and thereafter maintain intricate special-purpose test harnesses over the useful life of the software being supported.